Confused you are probably correct about your mother, but cult indoctrination could be playing a large part in it too.
I think you know that the kids will only suffer at the hands of your mother if he gets her way with them. Stick to your terms and make sure you are always present so cult indoctrination of the kids can be nipped in the bud or avoided if possible.